Assessing the Performance of Multiprogram Workloads Running on Multithreaded Hardware Is Difficult Because It Involves a Balance between Single-program Performance and Overall System

......Performance metrics are the foundation of experimental computer science and engineering. Researchers and engineers use quantitative metrics for assessing their new ideas and engineering progress. Obviously, adequate metrics are of primary importance to research progress, whereas inappropriate metrics can drive research and development in wrong or unfruitful directions. The recent trend toward multicore and many-core systems makes adequate performance metrics for assessing the performance of multithreaded computer systems running multiple independent programs essential. There are two key reasons for this need. First, as the number of on-chip cores increases exponentially according to Moore’s law, more multiprogram workloads will run on the hardware. Second, coexecuting programs affect each other’s performance through sharing of resources such as memory, off-chip bandwidth, and (potentially) on-chip caches and interconnection networks. In addition, programs can share resources within a single simultaneous multithreading (SMT) core. Quantifying single-program performance is well understood. Researchers have reached the consensus that the performance metric of choice for assessing a single program’s performance is its execution time. For single-threaded programs, execution time is proportional to CPI (cycles per instruction) or inversely proportional to IPC (instructions per cycle), provided that cycle time and dynamic instruction count are constant. However, for multithreaded programs, IPC and CPI are poor performance metrics. They can give a skewed performance picture if threads spend time in spin-lock loops (and other synchronization mechanisms) and idle loops in system code without making forward progress. For Stijn Eyerman Lieven Eeckhout